Nonetheless, their performance heavily depends on the specificity of the input text prompts, which requires skillful prompt template engineering. Instead, current approaches to prompt optimization learn the prompts through gradient descent, where the prompts are treated as adjustable parameters. However, these methods tend to lead to overfitting of the base classes seen during training and produce prompts that are no longer understandable by humans.

"secondary data" not reflected in their species labels [

To tackle this issue, we conducted the theoretical analysis to promote the effectiveness of contrast decoding.

In this paper, we propose a post-training quantization framework of large vision-language models (L VLMs) for efficient multi-modal inference. Conventional quantization methods sequentially search the layer-wise rounding functions by minimizing activation discretization errors, which fails to acquire optimal quantization strategy without considering cross-layer dependency.

Specifically, we observe that CLIP's visual encoder tends to

Additionally, a learnable component of the language prompts is trained using the available few-shot labeled data.